|
A beginner’s guide to reporting and tracking defects
By: Mitch Allen
BugTracking Basics
What Is a Bug Tracking System?
A bug tracking system is constructed as a
database. The system may be based on
an existing database such as Lotus Notes,
or it may use a built-in proprietary database.
The bug tracking system may be
homegrown or it may be a commercial
system. In any case, a bug is entered as a
record into this database, where it can be
assessed and tracked.
The user interface usually consists of
navigable views and forms for editing existing
bug reports or writing new ones.
Each user is assigned a unique login ID
and password. Whenever a user writes up
a bug, adds comments, or changes a status
field, a good tracking system will log the
ID of the person, creating or amending the
record. A great system will save such history
as read-only data so a true history of
a bug report can be maintained.
A good tracking system will give each
member of the team easy access to bugs
assigned to them. The design should
make it easy for the team members to
manage their specific bugs effectively
with the data sorted in many different
ways. A great system will give management
a variety of views and reports reflecting
the health of the product, so they
can assess the product situation on a
build-by-build basis.
Of course, some users of the system
are given more rights than others. For example,
most users aren’t allowed to
delete bugs. If a bug is a duplicate or entered
in error, it is simply marked as such
instead of being deleted.
One of the most important features of
a bug tracking system is keyword searching.
For example, let’s say a user finds a
crash bug while saving a file. The bug
tracking system should enable the user to
perform a keyword search on “crash and
file.” If a match isn’t found, the bug can
be entered. If existing bugs are found, the
user should examine them to see if the report
covers his circumstance.
The Bug Lifecycle
Let’s examine a bug’s lifecycle:
1. Someone finds a bug and enters it into
the system
2. The bug is assigned to a developer
3. The developer looks at the bug
When a bug is entered, the author usually
assigns it to a product category such as
“user interface” or “calc engine.”
Writing or entering a bug is also
known as opening a bug. Some systems
may default a bug to “New” and wait
for someone to open it. From my experience,
doing so is a wasted step.
Depending on how the database is
constructed, the bug may be assigned automatically
to a developer and QA person
according to the chosen category. It’s
up to the developer to look for open bugs
with her name on them. It is up to the development
manager to look for open, yet
unassigned bugs and assign them as soon
as possible. Once a bug is assigned, most
systems will send the developer an automatic
email notification. Bugs may be reassigned
if it is determined that the default
person is not the best person to fix
the problem. Sometimes, if a developer
sees an unassigned bug in her area, she
will take on the assignment and proceed
to work on it. A QA person should be
just as proactive if he sees unassigned
bugs in his area as well.
Once a developer takes ownership of
a bug, she assesses the situation and sets
a resolution field. The bug’s state is pending
after at least one person (usually the
developer) comments on the bug or sets
the resolution field. From this point on,
the bug remains pending until someone
either closes or verifies it.
A bug is verified when the original
steps no longer reproduce the problem
and someone in QA has reviewed the fix.
But this doesn’t mean that the bug can be
closed. The bug is usually kept in this
state until QA has done a thorough
check under all supported operating systems.
If verification fails, what happens to
the bug is a gray area. Some systems may
revert it back to “open,” or set some other
flag so it lands back at the feet of the
developer or development manager.
It is up to the QA person assigned
and/or the author of the bug to review
the resolution and decide if they agree.
QA usually has the responsibility of
closing a bug. This happens when either
they are satisfied that the bug has been
properly verified, or they agree that the
bug is no longer (or never was) an issue.
Resolutions
A resolution can be one of the following:
Fix—The developer thinks she has a
fix to the problem. A field indicating
which build the bug should be fixed in
will accompany this. The QA person assigned
the bug should wait for that build
to be released, then verify whether the fix
actually works. If it doesn’t work, this isn’t
a plot by the developer to fool anyone.
When a developer makes a fix, it
may be difficult for her to verify it until
the build is released.
Defer—A bug is usually deferred
when management or all parties agree
that the issue isn’t drastic, and can be put
off (deferred) until a later time.
Dupe (Duplicate)—This means that
someone else has already reported the
problem. When you first start on a project,
you will see this in your own bugs
more often than you would like. One
way to avoid starting out with many
dupes is by spending an afternoon reviewing
the bugs that are currently open.
Mastering your system’s keyword searching
capabilities is another way of avoiding
too many dupes.
A good system will have a field for the
original bug number. The person marking
the bug as a dupe should fill in the
number of the original to make it easy
for others to go back, review the original,
and see if they agree. It’s quite possible
that the person marking the dupe is assuming
too much. If you think the bug is
not a dupe, you should reset the resolution
field and note as such.
Need More Information (NMI)—
This means the developers don’t have
enough information to understand or reproduce
the problem. They should note
in the bug what they are looking for.
Not a Bug (NAB)—The developers
disagree that it’s a bug. Another way of
saying this is Working as Designed
(WAD).
Not Reproducible (NREP)—The bug
could not be reproduced. This may be either
because there is something strange
about your setup (you have a driver or
service pack that the developer is missing,
or vice versa), or the person looking
at the bug is trying it in a later build. The
bug may have been inadvertently fixed
since you encountered it in the last build.
However, if you can still reproduce it
on your machine with the later build,
you should disagree, reset the resolution,
and talk to the developer.
No Plan to Fix (NPTF)—The bug has
been recognized as a problem, but the developers
don’t plan to do anything about
it. You’ll see a lot of this as the product
approaches its ship date and the pressure
mounts. If you disagree with the resolution,
you may need to bring it to your
boss’s attention.
User Error—This is a polite way of
saying that the problem isn’t with the
product. User error is sometimes referred
to as UBD (User Brain Damage).
Bug Components
A typical bug report contains the following
components:
Bug Number—A bug report contains a
unique ID, so it can be tracked. Whenever
you tell anyone about a bug, the first thing
you’ll be asked for is the bug’s number or
ID. The number or ID allows the developer
to enter the database and look it up.
Short Description—The short description
may also be referred to as the
“title” of the bug. This is one of the most
important fields to get right. Filling it in
with “Product doesn’t work” will probably
incur the wrath of not only your
peers, but also your boss. When sifting
through a list of bugs, the short description
needs to be clear, concise, and to the
point. One example is “Crash on exit
only after save.” That will get their attention.
You will notice that the description
doesn’t need to be a complete sentence.
Product/Project Name—Versions of a
product are usually given code names.
For example, Lotus 1-2-3 version 3.0
was code-named “Godiva.” Dragon NaturallySpeaking
version 3.52 was codenamed
“Yukon.”
There was a time when code names
were used to keep outsiders guessing as
to what secret project people were working
on. But you will notice more often
that code names are being used in press
releases, along with the actual product
and version number.
Product Version—The product version
is the public designation for when
the product ships (such as Dragon NaturallySpeaking
3.52). Version numbers
usually include point-release identifiers
(3.01 versus 3.02). It’s usually a good
idea to keep the product version number
as a separate field in the bug report.
Some companies assign different project
names to different versions. If the point
release only contains one or two bug fixes,
some companies won’t bother assigning
different project names.
Build Number—Every time someone
(usually a developer or someone designated
as the build/release engineer) recompiles
the product and incorporates
the latest bug fixes, it is given a new build
number. The build in which a bug was
found should be noted in the bug report.
Fixed Build Number—When a developer
fixes (or hopes she has fixed) a bug,
she must note in the bug report which
build the fix will be in. The fix can’t be
verified as working until a copy with that
build number or higher is released.
Steps to Reproduce—Before you
write one line on how to reproduce a
bug, you should attempt to narrow any
extra steps that are not absolutely needed
to reproduce it. (If you’ve ever used a
debugger or watched a developer use a
debugger, you’ll know why.) The developer
might need to walk through every
line of code as it executes for each step.
For instance, clicking File/Save may
seem trivial, but that could launch a
very long sequence of events or large
loops that the developer needs to step
through or wait for.
Comments—Additional comments
Bug A bug is another term for describing a defect in a software program.
If you start the product and it crashes, that’s a bug. If you find a typo
in the Help system, that’s considered a bug too.
Bug Report You may have also heard the initials “SPR” (Software
Problem Report), which is the same as a bug report.
Bug Tracking Some people refer to a bug tracking system as a “defect
tracking system.” It’s the same thing. I’ve even seen it called a “nonconformity
reporting system.”
Build Every time Development recompiles the code, it is considered a
new “build.” Each build is assigned a number or an alphanumeric string indicating
some sort of sequence. It is imperative that for each bug written,
there is a note of which build the bug was found in, as well as which build
it was fixed in. (Because Web site development is more fluid, assigning a
build number may be difficult. Some companies may assign build numbers
to different components of a Web site rather than to the whole site itself.)
Development This can also include the documentation group. Both
parties are responsible for developing something (software versus content),
and they are usually lumped in the same category.
Product When I refer to a product, it can be either a software application
or a Web site.
QA, QE, SQA These are all the same thing. QA stands for Quality Assurance.
QE stands for Quality Engineering (usually used when a QA team
wants to be given the same respect as the developers). SQA stands for
Software Quality Assurance. SQA is often used on job boards to distinguish
software quality professionals from quality control officers in medical
and manufacturing industries.
Releasing a Build Before a product is shipped to customers or posted
to a public Web site, it goes through many builds that are released internally
for testing. This means that development has successfully compiled
the latest bug fixes and/or product updates and is releasing them to
the rest of the team. They may have run a simple test to verify basic functionality,
but that’s usually all that is guaranteed. It’s up to QA to verify
whether the build is healthy enough for general testing.
Shipping When I refer to shipping, I am talking about releasing a software
product to the public. It can also refer to posting a Web site to publicly
available servers.
Tester This is either an SQA professional or anyone else drafted to help
test a product.
And a few words about Regression I’d like to point out that
a “regression” is not a bug. If a developer tells you that they’ve “introduced
a regression”—well, that’s because they don’t want to come out and say
that they broke the product. Defining what regression really means is outside
not directly related to the bug should be
reserved for email.
Author—Most systems will fill this in
automatically, based on the login ID.
Operating System—This is very important.
If a developer can’t reproduce
your bug, it could be because she’s using
a different operating system or version.
Web Browser—If you are testing a
Web site, the browser you’re using is as
important as the operating system.
Category—Always try to define a category
for a bug because this helps in assigning
it. If someone thinks that you
chose the wrong category, that person
will most likely correct it.
Subcategory—Some categories may
not have a subcategory. If you think a
subcategory is appropriate, then you
should select it. You’ll need to pick one if
your system demands that a selection be
made. As with the category, if you make
an incorrect choice, someone else will
probably correct it.
Developer Assigned—The developer
field should contain a list of developers
to choose from. Some systems will assign
a developer by default, based on the category.
Other systems may leave this field
as “unassigned.” If a bug is left unassigned,
it’s usually the responsibility of a
manager or the developers to assign the
bug to the appropriate person.
You should be diligent about the bugs
you enter and make sure they don’t stay
unassigned for too long. If this is the
case, find out what your organization’s
protocol is for assigning unassigned
bugs. It may simply be that the mechanics
of automatic assignment are not in
place, allowing you to assign the bug
yourself, working either from a list or
common knowledge. If the assignment is
not appropriate, someone will correct it.
If you are a developer and a bug is erroneously
assigned to you, you should
assign it to someone else; assign it to
your boss and let him figure it out; or revert
it back to an unassigned state (if possible).
Whenever you change the assignment
of a bug, you should leave a simple
note as to why you’ve changed it so the
bug isn’t reassigned to you.
Developer Resolution—Development
sets this field after they have looked at
the bug.
QA Assigned—The QA field should
contain a list of QA engineers. As with
the developer field, some systems will assign
this field automatically. If it isn’t assigned
automatically, then a manager will
assign the field or QA people can assign
bugs in their area to themselves.
State—Open, Pending, Verified, and
Closed are the most common bug states.
Priority—Some bug systems allow a
priority field to be set. This may be a keyword
list that includes Urgent, High,
Medium, and Low, or a number whose
value is defined by the department. This
helps developers distinguish which bugs
to fix first.
Severity—This field is used to define
how severe a bug is. For example, a user
being able to log in and see someone else’s
credit card information may not be a
crash bug, but it’s still a severe problem.
History—A good bug system will
record all comments and field changes, as
in a read-only history field that anyone
can review, but not edit.
Attachments—These can include
bitmaps, files, anything you want. When
attaching anything to a bug, make sure it’s
in *.zip format. Quite often, attachments
stretch hardware limits and can slow down
the system. Many organizations insist on
storing attachments on public servers and
just leaving a reference in the bug.
Miscellaneous/Custom—Various departments
may have other fields added to
bug reports for their own use.
Tips for the Beginner
You may be new to SQA, but you probably
know that many people are rated on
how many bugs they find. There is an implied
time limit for how long you should
spend chasing down one bug, so you
shouldn’t let one bug consume your life.
If you can’t nail it down, bite the bullet,
write a report where you state as much as
you can, and move on. If it’s a nasty crash
bug, tell your boss and work out whether
it requires more attention. If so, enlist the
aid of others to track it down.
Because of this rating system, some
people new to QA may get the urge to
“stretch” a bug by making several bugs
out of one. An example of this would be
writing a bug where the user enters some
text, noting the product crashes, then
writing up the same bug, but entering
different text. Don’t do this. You’ll just
waste everyone’s time, and your extra
bugs will probably be flagged as dupes.
Before writing up a bug, ask Development
what’s fair and what isn’t. “Fair
game” refers to areas where it is fair for
QA to write bugs. If Development tells
you that the File Import routine is still
under construction and not guaranteed
to work, this isn’t fair game. Writing up
bugs against this area is another waste of
time and can make people angry.
Another tip is to review the bug system
as a whole. Examine the trends and
try to get a feel for what areas are vulnerable
and need more testing. If the bug
count increases over time (builds), this is
a bad thing. If the bug count decreases, it
could mean that the developers have kidnapped
the QA team, but it’s more likely
to mean the product is becoming more
stable (and approaching a date when it
can ship/post).
One more thing to look out for is developers
going wild by marking everything
(including crash bugs) as NPTF
(No Plan To Fix), NAB/WAD (Not A
Bug/Working As Designed), or NREP
(Not Reproducible).
Management’s Role
The first thing a manager should do is to
make sure that all bugs are assigned. But
since managers are usually stuck in meetings
all day, team members need to be
proactive by taking on the unassigned
bugs themselves.
Managers should do everything they
can to squeeze data out of the bug tracking
system—especially graphs. Although
I could dedicate a whole article to the
types of reports and graphs that are
needed, you basically need to chart how
the product is doing over time. Time can
best be mapped out by build number.
Final Thoughts
Nothing in this article is cast in stone.
I’ve tried to provide an overview of a
generic system, based on my experience
of using and building bug tracking systems.
As you learn your company’s system,
you may find that items and procedures
are a little different.
Regardless of the system you are using,
mastering its intricacies can help you to
better manage your workload, generate
comprehensive reports, and track progress
for yourself and your department.
Other Resource
... to read more articles, visit http://sqa.fyicenter.com/art/
|